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Report  Title 

Advanced  Methods  for  Representing  and  Processing  Hyperspectral  Image  Data 

ABSTRACT 

We  have  extended  our  3D  spectral/ spatial  Gabor  representation  to  consider  the 
effects  of  three-dimensional  scene  structure  in  hyperspectral  images.  We 
have  shown  that  traditional  spectral/spatial  models  lead  to  ambiguities  when 
classifying  image  regions  due,  in  part,  to  changes  that  occur  as  the 
environmental  conditions  change.  Our  new  models  characterize  the  variation 
of  vectors  that  are  derived  using  spectral/ spatial  features  as  the  scene 
conditions  change.  We  have  shown  that  these  models  improve  on  the  properties 
of  standard  techniques.  The  utility  of  this  approach  has  been  demonstrated 
using  thousands  of  hyperspectral  image  regions  that  have  been  generated  over 
a  broad  range  of  conditions.  We  have  also  modeled  the  effect  on  hyperspectral 
image  sequences  of  ballistic  impacts.  We  have  shown  that  the  multiband  Gabor 
representation  allows  extraction  of  image  properties  that  can  be  used  to 
estimate  properties  of  the  impact.  We  have  also  applied  the  models  to  human 
skin  in  near-infrared  images  and  demonstated  the  potential  for  tissue 
classification. 
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Scientific  Progress 

We  have  implemented  a  method  for  generating  a  3D  Gabor  filter  bank  for  the 
representation  of  hyperspectral  image  data.  We  have  used  this  approach  to 
generate  compact  expressive  models  for  hyperspectral  image  regions.  We  have 
also  derived  methods  for  selecting  filters  that  emphasize  the  most  significant 
spectral/spatial  differences  between  classes.  We  have  shown  that  the  new 
approach  can  be  used  for  accurate  clutter  classification  in  AVIRIS 
hyperspectral  images  using  a  small  number  of  features.  We  have  extended  this 
approach  for  the  modeling  of  environmental  effects  on  images  of  3D  structures 
by  using  subspaces  of  multidimensional  Gabor  features.  These  models  have  been 
validated  by  generating  several  thousand  images  of  3-D  scenes  using  DIRSIG. 

We  have  also  performed  detailed  comparisons  of  the  new  models  and  methods  with 
previous  approaches.  Our  researchers  have  also  developed  new  methods  for  the 
realistic  synthesis  of  spectral/spatial  data  using  an  iterative  approach.  We 
are  also  modeling  the  thermal  properties  of  tissue  and  changes  that  occur  in 
response  to  ballistic  impacts.  These  models  have  been  used  to  synthesize 
hyperspectral  image  sequences  that  demonstrate  the  thermal  response  of  tissue 
to  ballistic  impacts.  Banks  of  multi-dimensional  Gabor  filters  have  also  been 
used  to  extract  properties  of  the  images  that  allow  estimation  of  properties  of 
the  impact.  In  addition,  we  are  considering  the  use  of  skin  models  for 
invariant  hyperspectral  face  recognition.  We  have  shown  that  the  models 
improve  performance  for  the  invariant  recognition  of  faces  in  three  dimensions. 
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1  Objectives 

Hyperspectral  imaging  (HSI)  sensors  provide  the  opportunity  to  address  a  wide  range  of 
applications.  The  large  number  of  spectral  bands  obtained  by  these  instruments  makes  it 
possible  to  use  spectral  information  to  perform  image  analysis  tasks  that  are  not  feasible  using 
other  sensors.  In  this  project  we  have  developed  innovative  models  for  HSI  data.  The  new 
models  utilize  multi-dimensional  Gabor  functions  for  the  extraction  of  spectral/spatial 
information  from  images  at  different  orientations  and  scales.  These  functions  provide  an 
efficient  means  of  sampling  a  multi-dimensional  frequency  domain  representation  of  HSI  data. 

In  order  to  reduce  the  dimensionality  of  the  data  we  have  developed  general  methods  for  the 
optimal  selection  of  multi-dimensional  Gabor  functions  that  capture  the  most  significant 
spectral/spatial  information  for  a  given  application.  We  have  also  considered  the  use  of 
multi-dimensional  Gabor  filters  for  modeling  time  sequences  of  hyperspectral  images.  This 
approach  is  being  used  for  the  modeling  and  analysis  of  kinetic  energy  effects  on  mammal 
tissues  against  ballistic  impacts.  HSI  sensors  have  the  unique  capability  of  obtaining  under-skin 
thermal  characteristics  of  the  ballistic  impacts  on  mammal  tissues.  The  focus  is  on  the  analysis 
and  understanding  of  hyperspectral  image  data  and  the  development  of  rigorous  mathematical 
models  to  characterize  ballistic  impacts.  The  proposed  approach  can  also  be  applied  to  the 
study  of  blunt  trauma  and  impulse  loading.  The  generality  of  our  approach  enables  the  new 
methods  to  advance  the  state-of-the-art  for  other  application  areas  such  as  target  detection  and 
clutter  classification.  Specific  objectives  for  this  project  are  listed  below. 

Objective  1:  Develop  methods  to  utilize  3D  and  4D  Gabor  filters  to  extract 
spectral/spatial/temporal  information  from  hyperspectral  images  and  image  sequences. 

Objective  2:  Develop  optimized  methods  for  reducing  the  dimensionality  of  the  Gabor 
representation. 


Objective  3:  Apply  the  new  approach  to  applications  including  the  analysis  of  ballistic 
impacts,  target  detection,  face  recognition,  and  clutter  classification. 


Objective  4:  Transfer  the  new  models  and  methods  to  DoD  users. 


2  Approach 

The  spectral/spatial  DFT  provides  a  general  model  for  spectral/spatial  image  structure  and  can  be 
inverted  to  recover  the  radiance  domain  data.  However,  the  cost  of  this  generality  is  the  size  of 
the  representation.  Thus,  spectral/spatial  models  with  fewer  parameters  are  desirable.  Markov 
random  fields  (MRFs)  have  been  used  for  hyperspectral  image  modeling.  These  models  provide 
a  compact  image  representation  by  restricting  the  dependence  of  the  data  at  a  pixel  to  a  defined 
set  of  neighboring  pixels.  However,  we  have  demonstrated  limitations  of  MRFs  when  applied 
to  real  data  and  other  studies  have  shown  empirical  limitations  of  the  models  for  various 
applications.  Filter-based  approaches  have  also  been  used  to  generate  compact  models  for 
multispectral  images.  However,  these  approaches  apply  the  filters  to  individual  spectral  bands 
and  do  not  exploit  the  three-dimensional  spectral/ spatial  structure  of  the  data. 

The  spectral/spatial  DFT  represents  an  image  region  using  a  dense  sampling  in  the  frequency 
domain.  An  alternative  approach  is  to  represent  the  DFT  by  its  projection  onto  a  set  of 
functions  that  capture  specific  orientation,  scale,  and  spectral  attributes  of  the  image  data.  For 
this  purpose,  we  use  a  new  representation  for  spectral/spatial  data  using  three-dimensional  Gabor 
functions.  Two-dimensional  Gabor  functions  have  been  used  to  represent  scale  and  orientation 
information  in  single -band  image  data  for  several  applications.  These  functions  achieve  optimal 
joint  localization  in  space  and  spatial  frequency.  Previous  work  has  applied  2D  Gabor  functions 
to  multiband  images  band-by-band  or  by  considering  opponent  relationships  between  pairs  of 
bands.  Two-dimensional  Gabor  functions  have  also  been  used  as  the  mother  function  to  define 
a  wavelet  decomposition  for  2D  images.  Other  work  has  used  wavelets  to  represent  the  spectral 
dimension  in  hyperspectral  data.  However,  a  general  approach  to  representing  spectral/spatial 
image  data  requires  the  use  of  3D  basis  functions.  We  are  also  considering  the  use  of  4D  Gabor 
functions  for  the  analysis  of  time  sequences  of  hyperspectral  images. 

3D  Gabor  filters  allow  for  a  large  number  of  spectral/spatial  quantities  to  be  used  to  represent  an 
image  region.  The  performance  and  efficiency  of  algorithms  that  use  this  representation  can  be 
improved  if  methods  are  available  to  reduce  the  dimensionality  of  the  model.  A  standard  goal  in 
signal  processing  is  to  generate  filtered  images  that  can  be  used  to  reconstruct  an  approximation 
to  an  input  image.  This  leads  to  a  set  of  filters  that  minimizes  the  reconstruction  error. 

However,  for  applications  such  as  characterization  and  classification  the  primary  goal  is  to  find 
filters  that  emphasize  the  most  significant  information  in  a  scene.  Thus,  we  have  derived  new 
methods  for  selecting  filters  that  are  optimized  for  this  goal. 


3  Scientific  Barriers 

This  research  is  the  first  attempt  to  use  3D  and  4D  Gabor  filters  to  represent  hyperspectral  image 
data.  Thus,  there  are  a  number  of  difficult  scientific  challenges  that  have  been  addressed.  For 
example,  there  are  important  issues  related  to  the  relative  scaling  of  the  spectral,  spatial,  and 
temporal  frequency  dimensions  that  must  be  considered.  In  multiple  dimensions  there  are  a 


large  number  of  correlations  that  can  be  taken  into  account.  New  methods  have  been  developed 
for  discovering  and  exploiting  these  correlations.  We  have  also  studied  the  state-of-the-art  in 
modeling  the  thermal  properties  of  tissue  to  understand  the  limitations  of  available  models.  In 
addition,  we  have  been  limited  in  our  ability  to  obtain  real  hyperspectral  data  showing  tissue 
response  to  ballistic  impacts.  There  are  also  a  large  number  of  details  that  had  to  be  addressed 
for  various  application  areas. 

4  Significance 

In  this  project  we  have  developed  innovative  models  for  spectral/spatial/temporal  data  that  will 
lead  to  significant  advancements  in  methods  for  automated  image  exploitation.  The  new  models 
utilize  three  and  four-dimensional  Gabor  functions  for  the  extraction  of  spectral/spatial/temporal 
information  from  images  at  different  orientations  and  scales.  These  functions  achieve  optimal 
joint  localization  in  space  and  frequency  and  provide  an  efficient  means  of  sampling  a 
high-dimensional  frequency  domain  representation  of  HSI  data.  We  have  used  physical  models 
to  develop  a  mathematical  structure  for  the  image  models  that  is  invariant  to  complex  changes  in 
the  environmental  conditions  in  the  presence  of  three-dimensional  scene  structure.  In  order  to 
reduce  the  dimensionality  of  the  data  we  have  developed  general  methods  for  the  optimal 
selection  of  three-dimensional  Gabor  functions  that  capture  the  most  significant  spectral/spatial 
information  for  a  given  dataset  and  application.  We  have  demonstrated  the  significance  of  the 
new  approach  by  detailed  comparison  with  previous  methods.  The  generality  of  the  new 
methods  has  been  shown  by  considering  several  applications  which  include  the  characterization 
of  ballistic  impacts,  target  detection,  face  recognition,  and  clutter  characterization. 

5  Accomplishments 

We  have  implemented  a  method  for  generating  a  3D  Gabor  filter  bank  for  the  representation  of 
hyperspectral  image  data.  We  have  used  this  approach  to  generate  compact  expressive  models 
for  hyperspectral  image  regions.  We  have  also  derived  methods  for  selecting  filters  that 
emphasize  the  most  significant  spectral/spatial  differences  between  classes.  We  have  shown 
that  the  new  approach  can  be  used  for  accurate  clutter  classification  in  AVIRIS  hyperspectral 
images  using  a  small  number  of  features.  We  have  extended  this  approach  for  the  modeling  of 
environmental  effects  on  images  of  3D  structures  by  using  subspaces  of  multidimensional  Gabor 
features.  These  models  have  been  validated  by  generating  several  thousand  images  of  3-D 
scenes  using  DIRSIG.  We  have  also  performed  detailed  comparisons  of  the  new  models  and 
methods  with  previous  approaches.  Our  researchers  have  also  developed  new  methods  for  the 
realistic  synthesis  of  spectral/spatial  data  using  an  iterative  approach.  We  have  also  modeled  the 
thermal  properties  of  tissue  and  changes  that  occur  in  response  to  ballistic  impacts.  These 
models  have  been  used  to  synthesize  hyperspectral  image  sequences  that  demonstrate  the  thermal 
response  of  tissue  to  ballistic  impacts.  Banks  of  multi-dimensional  Gabor  filters  have  also  been 
used  to  extract  properties  of  the  images  that  allow  estimation  of  properties  of  the  impact.  In 
addition,  we  have  considered  the  use  of  skin  models  for  invariant  hyperspectral  face  recognition. 
We  have  shown  that  the  models  improve  performance  for  the  invariant  recognition  of  faces  in 
three  dimensions. 


6  Collaborations 


We  have  presented  results  at  conferences  and  have  interacted  with  personnel  at  ARL,  NVESD, 
ECBC,  ARDEC  and  other  labs  with  the  goal  of  developing  collaborations  and  leveraged  funding. 
Dr.  Healey  also  led  a  session  on  applications  at  the  Workshop  on  Distributed  Video  Sensor 
Networks  at  UC  Riverside  and  delivered  a  plenary  talk  at  the  annual  SPIE  Photonics  Conference 
in  San  Diego. 

7  Scientific  Accomplishments 

We  have  developed  innovative  spectral/spatial/temporal  models  for  HSI  data  that  support  current 
needs  as  well  as  enable  future  research.  Methods  for  ensuring  invariance  to  environmental 
variability  in  3D  scenes  allow  the  models  to  be  used  over  a  broad  range  of  conditions  and 
dimensionality  reduction  techniques  have  been  used  to  allow  the  models  to  be  applied  efficiently. 
The  new  models  have  been  used  to  develop  automated  estimation  and  classification  tools  that 
address  several  application  areas  including  the  characterization  of  ballistic  impacts,  target 
detection,  clutter  characterization,  and  face  recognition.  We  have  demonstrated  the 
performance  of  these  tools  over  a  range  of  data.  We  expect  that  the  generality  of  our  approach 
will  enable  the  new  methods  to  advance  the  state-of-the-art  for  several  application  areas  and  to 
support  missions  involving  a  wide  range  of  HSI  sensors  and  imaging  configurations.  This 
project  has  also  trained  graduate  students  in  areas  of  interest  to  the  DoD. 

8  Conclusions 

We  have  developed  three-dimensional  Gabor  filters  as  a  tool  for  modeling  spectral/spatial 
information  in  multiband  images.  These  filters  provide  an  efficient  mechanism  for  sampling  a 
frequency  domain  representation  for  multiband  image  data.  We  have  also  developed  methods 
for  selecting  the  Gabor  filters  that  provide  the  most  discriminatory  power  for  a  given  application. 
The  new  approach  has  been  applied  to  several  application  areas  that  include  the  characterization 
of  ballistic  impacts,  target  detection,  face  recognition,  and  clutter  characterization. 


